Method for detecting particles using structured illumination

ABSTRACT

A particle detection method detects presence and location of particles on a target using measured signals from a plurality of structured illumination patterns. The particle detection method uses measured signals obtained by illuminating the target with structured illumination patterns to detect particles. Specifically, the degree of variation in these measured signals in raw images is calculated to determine whether a particle is present on the target at a particular area of interest.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/262,429 filed on Jan. 30, 2019 which claims the benefit of U.S.Provisional Patent Application No. 62/624,071 filed on Jan. 30, 2018,which is incorporated by reference herein in its entirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates generally to the field of opticalmicroscopy imaging which uses structured or selective illumination orexcitation and, more specifically, to a method of detecting particlesusing measured signals from structured illumination patterns.

2. Description of the Related Art

Synthetic Aperture Optics (SAO) imaging refers to an optical imagingmethod in which a series of patterned or structured light patterns areused to illuminate the imaging target in order to achieve resolutionbeyond what is set by physical constraints of the imaging apparatus suchas the lens and the camera. In SAO, an imaging target is selectivelyexcited in order to detect the spatial frequency information of thetarget. Since there is a one-to-one relationship between the frequency(or Fourier) domain and the object (or target) domain, SAO canreconstruct the original imaging target by obtaining its spatialfrequency information.

FIG. 1A illustrates a conventional SAO method, and FIG. 1B illustrates aconventional SAO system. Referring to FIGS. 1A and 1B together, inconventional SAO, selective excitation (or illumination) 104 is appliedto an imaging target 102, and the light scattered or fluoresced from theimaging target 102 is captured by optical imaging 106. The imagingtarget 102 can be composed of micro-particles in a randomly or regularlydistributed pattern. Selective excitation (or illumination) 104 may beapplied to the imaging target 102 by an illumination apparatus (notshown in FIGS. 1A and 1B) that is configured to cause interference 122of two or more light beams 131, 132 on the imaging target 102. Theexcitation is selective or patterned, unlike uniform illumination usedin conventional optical imaging techniques. For example, two beams 131,132 may overlay or interfere on an imaging-target plane 102 to produce atwo-dimensional (2D) sinusoidal excitation pattern.

FIG. 1C illustrates an example of a selective excitation pattern in thespatial domain and the frequency domain. Referring to FIGS. 1B and 1C,the exemplary selective excitation pattern 140 in the spatial domain isgenerated by interference of two beams 131, 132 on the imaging-targetplane 102, resulting in a 2D sinusoidal excitation pattern. The angle(φ) between the two beams 131, 132 determines the pitch 143 of thepattern, which represents the spacing or periodicity of 2D sinusoidalfringe pattern 140. More specifically, the pitch 143 is substantiallyinversely proportional to sin(φ). The orientation ϕ of the patternrepresents the amount of angular rotation of the 2D sinusoidal fringes140 compared to its reference pattern, which in this example of FIG. 1Cis shown as a 2D sinusoid comprised of vertical lines, although adifferent reference pattern such as a 2D sinusoid comprised ofhorizontal lines can also be used as the reference pattern. Inmathematical terms, the orientation ϕ can be described as follows: if uis the normal vector of the plane formed by the two beams 131, 132 andif the projected vector of u on the imaging plane 102 is called v, thenthe orientation ϕ of the sinusoidal pattern 140 is the angularorientation of the vector v with respect to the frame of reference. The“phase” of the pattern is the periodic position of the 2D sinusoid withrespect to the frame of reference. The range of the phases of the 2Dsinusoid excitation pattern will be a value between 0 and 2π. Thedifferent phases can be obtained by changing optical path length of onebeam.

As shown in FIG. 1C, the 2D sinusoid excitation pattern in the spatialdomain can be shown as a conjugate pair k_(i), k_(i)′ in thecorresponding frequency domain (k-space). Each conjugate pair in thek-space corresponds to the pitch 143 and orientation ϕ of thecorresponding 2D sinusoid pattern. The pitch 143 of the 2D sinusoidpattern 140 is determined by the radial distance r of the k-spacepoint—more precisely, the pitch 143 is substantially the inverse of theradial distance r in the frequency domain. The orientation ϕ is theangle ϕ of the k-space points in a radial coordinate system in thefrequency domain. Thus, a number of different excitation patterns may begenerated by changing the pitch 143 of the 2D sinusoid pattern (or theangle (φ) between the two beams 131, 132) and changing the orientation ϕof the 2D sinusoid pattern, with each different pair of pitch 143 andorientation ϕ of the 2D sinusoid pattern in the spatial domaincorresponding to a different conjugate pair (radial distance r andorientation ϕ) in the k-space (frequency) domain.

Referring back to FIGS. 1A and 1B, the excited target 102 emits signals(or photons), and the signals are captured in optical imaging system 106including an objective lens 124 and an imaging sensor (or imager) 126.The emitted signal will have a wavelength λ_(E). The objective lens hasmagnification (Mag) and a numerical aperture NA=n×sin θ, where n is theindex of refraction of the medium in which the lens 124 is placed and θis the half-angle of the maximum cone of light that can enter or exitthe lens 124. Typically, the imaging sensor 126 can be a charge-coupleddevice (CCD), complementary metal-oxide-semiconductor (CMOS) imagesensor, or any other photon detectors in a matrix or array formatincluding a plurality of pixels m. Note that, in some applications, theemitted signals from the target 102 may be directly captured by theimager 126 without going through the objective lens 124.

Then, it is determined 108 whether the images corresponding to all thephases of the 2D sinusoid excitation pattern were obtained. If imagescorresponding to all the phases of the 2D sinusoid excitation patternwere not obtained in step 108, the excitation phase is changed 114 andsteps 104, 106, 108 are repeated for the changed excitation phase. Ifimages corresponding to all the phases of the 2D sinusoid excitationpattern were obtained in step 108, then it is determined 110 whether theimages corresponding to all the 2D sinusoid selective excitationpatterns were obtained. If images corresponding to all the 2D sinusoidselective excitation patterns were not obtained in step 110, theexcitation pattern is changed by using a different spatial frequency(e.g., changing the pitch 143 and the orientation ϕ of the 2D sinusoidpattern) and steps 104, 106, 108, 114 are repeated for the nextselective excitation pattern.

If images corresponding to all the 2D sinusoid selective excitationpatterns were obtained in step 110, then finally the captured images aresent to a computer for SAO post processing 112 and visualization. Inconventional imaging, the resolution of the SAO imaging system isdetermined by the numerical aperture NA of the lens 124, the wavelengthλ_(E) of the emitted light, and the pixel size. In contrast, in SAOimaging, the resolution of the imaging system is beyond what can beachieved by the numerical aperture NA of the lens 124, the wavelengthλ_(E) of the emitted light, and the pixel size. Thus, as shown in FIG.1B, the images captured through steps 104, 106, 108 of FIG. 1A are rawimages RI_(i) with a resolution lower than (insufficient for) theresolution needed to resolve the objects on the imaging target 102.However, multiple sets of the lower resolution raw images RI_(i) arecaptured for different excitation phases and spatial frequencies(excitation patterns) to obtain the complete raw image set 128, whichthen goes through SAO post-processing 112 to synthesize the final imageFI that has a resolution higher than the resolution of the raw imagesRI_(i). The resolution of the final image FI obtained by SAOpost-processing is sufficient for resolution of the objects on theimaging target 102. The methodology for SAO post-processing 112 forsynthesizing high resolution images FI from lower resolution raw imagesRI_(i) is well known. Raw images RI_(i) are converted into k-spaceinformation of the high resolution images FI, and this information isFourier transformed to synthesize or reconstruct the high resolutionimages FI. For example, one example of the SAO post-processingmethodology can be found in U.S. Pat. No. 6,016,196, issued on Jan. 18,2000 to Mermelstein, entitled “Multiple Beam Pair Optical Imaging,”which is incorporated by reference herein.

Applying SAO to DNA (deoxyribonucleic acid) or RNA (ribonucleic acid)sequencing presents a number of challenges. The term “nucleic acid”herein includes both DNA and RNA. In DNA or RNA sequencing, singlemolecule or amplified clones of a DNA template (collectively referred toas “microparticle”) are immobilized onto a planar substrate. The arrayof microparticles then goes through multiple cycles of chemical reactionand optical detection. FIGS. 2A, 2B, and 2C illustrate different typesof individual sequencing microparticles that can be used for DNAsequencing. FIG. 2A illustrates an individual microparticle 202 formedby a 1-micrometer diameter bead 208 covered with clonal DNA molecules210 that have been previously amplified by a water-in-oil emulsion PCRtechnique. The bead 208 is attached directly to the substrate 204 influid 206. FIG. 2B illustrates an individual microparticle 202 as acluster of clonal DNA molecules 210 attached to the substrate 204 andplaced in fluid 206. The DNA molecules 210 have been previouslyamplified by a bridge amplification technique. FIG. 2C illustrates anindividual microparticle as a single DNA molecule 210 attached to thesubstrate 204 and placed in fluid 206. The single DNA molecule 210 issequenced without amplification.

The distribution of DNA microparticles can be random or regular. FIGS.3A and 3B illustrate some examples of the distribution of DNAmicroparticles. If Δx is defined to be the spatial resolution of animaging system (i.e., Δx is the minimum distance of two point objectsthat can be resolved by the imaging system), Δx is typically designed tobe about half of the distance between adjacent microparticles 202 (seeFIG. 3A). In DNA sequencing applications, it is highly desirable for anoptical imaging system to achieve both high resolution and high scanningspeed at the same time. SAO imaging is promising since it can image alarge area using a low magnification lens and camera without sacrificingresolution. The resolution of SAO imaging is obtained from the highresolution illumination patterns and post-processing. However, SAOrequires selective excitation to be repeated for a number of selectiveexcitation patterns. Conventional SAO imaging uses a large number of SAOexcitation patterns, often including many redundant or even irrelevantillumination patterns. The number of excitation patterns in conventionalSAO is merely determined based on the hardware architecture of theillumination system, without regard to other factors. The large numberof excitation patterns in conventional SAO makes it impractical for usein DNA sequencing, as conventional SAO does not offer the cost andthroughput benefit in DNA sequencing compared to conventional optics.Also, conventional SAO hardware is large, complex, difficult to scale,and mechanically and thermally unstable, requiring large space andextremely careful control of temperature and mechanical vibration forcontinuous run, making it particularly impractical for use in DNAsequencing which requires repeated, continuous runs of SAO over a verylarge number of DNA microparticle arrays.

Typically, given the final reconstructed image, a detection systemestimates whether particles are present on a plurality of regions on atarget by generating a set of reconstruction estimates based on theintensity values of the reconstructed image. For example, areconstruction estimate for a pixel location of the reconstructed imagemay indicate whether a particle is present at a corresponding region ofthe target by comparing the intensity value for the pixel location to apredetermined threshold. However, it is often difficult to detectparticles with high accuracy in this manner due to, for example, thetexture of the target that results in a noisy reconstruction image.

SUMMARY OF THE INVENTION

Embodiments of the present invention include a method for syntheticaperture optics (SAO) that minimizes the number of selective excitationpatterns used to illuminate the imaging target based on the target'sphysical characteristics corresponding to spatial frequency content fromthe illuminated target and/or one or more parameters of the opticalimaging system used for SAO. Embodiments of the present invention alsoinclude an SAO apparatus that includes a plurality of interferencepattern generation modules that are arranged in a half-ring shape.

In one embodiment, an SAO method comprises illuminating the targetincluding one or more objects with a predetermined number (N) ofselective excitation patterns, where the number (N) of selectiveexcitation patterns is determined based upon the objects' physicalcharacteristics corresponding to spatial frequency content from theilluminated target, optically imaging the illuminated target at aresolution insufficient to resolve the objects on the target, andprocessing optical images of the illuminated target using information onthe selective excitation patterns to obtain a final image of theilluminated target at a resolution sufficient to resolve the objects onthe target. In another embodiment, the number (N) of selectiveexcitation patterns corresponds to the number of k-space sampling pointsin a k-space sampling space in a frequency domain, with the extent ofthe k-space sampling space being substantially proportional to aninverse of a minimum distance (Δx) between the objects that is to beresolved by SAO, and with the inverse of the k-space sampling intervalbetween the k-space sampling points being less than a width (w) of adetected area captured by a pixel of a system for said optical imaging.

In another embodiment, an SAO apparatus comprises a plurality ofinterference pattern generation modules (IPGMs), with each IPGMconfigured to generate a pair of light beams that interfere to generatea selective excitation pattern on the target at a predeterminedorientation and a predetermined pitch, and with the IPGMs arranged in ahalf-ring shape. The SAO apparatus also comprises an optical imagingmodule configured to optically image the illuminated target at aresolution insufficient to resolve the objects on the target. Theoptical image of the illuminated target is further processed usinginformation on the selective excitation patterns to obtain a final imageof the illuminated target at a resolution sufficient to resolve thetarget. The number of IPGMs is equal to the number of selectiveexcitation patterns used for performing SAO on the target. The IPGMs maybe placed substantially symmetrically on a monolithic structure that hasthe half-ring shape.

According to various embodiments of the present invention, an optimized,minimum number of excitation patterns are used in SAO, thereby enablingSAO to be used with applications such as DNA sequencing that requiresmassive parallelization of SAO imaging in a short amount of time to makeDNA sequencing with SAO commercially feasible. Thus, dramatic increaseof throughput and reduction of cost for DNA sequencing can be achievedby using SAO according to the present invention.

Embodiments of the present disclosure also include a method fordetecting particles on a target. Embodiments of the present disclosurealso include a system for detecting particles on a target.

In one embodiment, a particle detection method comprises illuminatingthe target with a plurality of structured illumination patterns that areeach characterized by a spatial frequency and an illumination phase,generating a plurality of raw images of the target by measuring opticalsignals from the illuminated target, each raw image including at leastone raw intensity value obtained from measurements of the targetilluminated with a corresponding structured illumination pattern, andfor each of one or more regions of the target, generating a firstestimate that indicates whether a particle is present at said each ofone or more regions of the target. Generating the first estimatecomprises for said each of one or more regions of the target,determining a modulation score by combining a set of raw intensityvalues from the plurality of raw images, the modulation score indicatinga degree of variation in the set of raw intensity values in said each ofone or more regions of the target, and generating the first estimate forsaid each of one or more regions of the target by comparing themodulation score for the region to a first threshold

In another embodiment, a system for detecting particles on a targetcomprises a plurality of illumination modules configured to illuminatethe target with a plurality of structured illumination patterns that areeach characterized by a spatial frequency and an illumination phase. Thesystem also comprises an optical imaging module configured to generate aplurality of raw images of the target by measuring optical signals fromthe illuminated target, each raw image including at least one rawintensity value obtained from measurements of the target illuminatedwith a corresponding structured illumination pattern. The system alsocomprises a detection module configured generate, for each of one ormore regions of the target, a first estimate that indicates whether aparticle is present at said each of one or more regions of the target,wherein for said each of one or more regions of the target. Thedetection module is further configured to determine a modulation scoreby combining a set of raw intensity values from the plurality of rawimages, the modulation score indicating a degree of variation in the setof raw intensity values in said each of one or more regions of thetarget, and generate the first estimate for said each of one or moreregions of the target by comparing the modulation score to a firstthreshold.

The features and advantages described in the specification are not allinclusive and, in particular, many additional features and advantageswill be apparent to one of ordinary skill in the art in view of thedrawings, specification, and claims. Moreover, it should be noted thatthe language used in the specification has been principally selected forreadability and instructional purposes, and may not have been selectedto delineate or circumscribe the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the embodiments of the present invention can be readilyunderstood by considering the following detailed description inconjunction with the accompanying drawings.

FIG. 1A illustrates a conventional SAO method.

FIG. 1B illustrates a conventional SAO system.

FIG. 1C illustrates an example of a selective excitation pattern in thespatial domain and the frequency domain.

FIGS. 2A, 2B, and 2C illustrate different types of individual sequencingmicroparticles that can be used for DNA sequencing.

FIGS. 3A and 3B illustrate some examples of the distribution of DNAmicroparticles.

FIG. 4 illustrates an SAO method, according to one embodiment.

FIG. 5A illustrates the k-space sampling points (selective excitationpatterns) used in SAO, according to one embodiment.

FIG. 5B illustrates the selection of the k-space sampling interval usedin SAO, according to one embodiment.

FIG. 5C illustrates using selective excitation patterns corresponding tok-space sampling points within a circular region, according to oneembodiment.

FIG. 5D illustrates reducing the number of k-space sampling points bysparse k-space sampling, according to one embodiment.

FIG. 6A illustrates how aliasing occurs in SAO by use of a pixel fieldof view (PFOV) smaller than the detected area, according to oneembodiment.

FIG. 6B illustrates how the actual signal at a pixel of an imagingsystem may be determined by unfolding the measured signal at the pixelto remove aliasing, according to one embodiment.

FIG. 6C illustrates a method of unfolding the measured signal at thepixel to remove aliasing, according to one embodiment.

FIG. 7A illustrates a structured illumination apparatus for selectivelyexciting the microparticles, according to one embodiment.

FIG. 7B illustrates the arrangement of the illumination patterngeneration modules in a half-ring structure, according to oneembodiment.

FIG. 7C illustrates the internal structure of an illumination patterngeneration module, according to one embodiment.

FIG. 7D illustrates the internal structure of an illumination patterngeneration module, according to another embodiment.

FIG. 8 illustrates a particle detection method, according to oneembodiment.

FIG. 9 illustrates a particle detection method, according to anotherembodiment.

FIG. 10 illustrates examples of particle detection methods that wereperformed on a tissue section target area, according to one embodiment.

FIG. 11 illustrates examples of particle detection methods that wereperformed on single molecule mRNA FISH (fluorescence in situhybridization) samples, according to one embodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

The Figures (FIG.) and the following description relate to preferredembodiments of the present invention by way of illustration only. Itshould be noted that from the following discussion, alternativeembodiments of the structures and methods disclosed herein will bereadily recognized as viable alternatives that may be employed withoutdeparting from the principles of the claimed invention.

Reference will now be made in detail to several embodiments of thepresent invention(s), examples of which are illustrated in theaccompanying figures. It is noted that wherever practicable similar orlike reference numbers may be used in the figures and may indicatesimilar or like functionality. The figures depict embodiments of thepresent invention for purposes of illustration only. One skilled in theart will readily recognize from the following description thatalternative embodiments of the structures and methods illustrated hereinmay be employed without departing from the principles of the inventiondescribed herein.

Synthetic aperture optics (SAO) imaging method according to variousembodiments of the present invention minimizes the number of selectiveexcitation patterns used to illuminate the imaging target, based on thetarget's physical characteristics corresponding to spatial frequencycontent from the illuminated target and/or one or more parameters of theoptical imaging system used for SAO. Embodiments of the presentinvention also include an SAO apparatus that is optimized to perform theSAO method according to the present invention. The SAO apparatusincludes a plurality of interference pattern generation modules that arearranged in a half-ring shape, each of which generates one selectiveexcitation pattern for SAO.

Turning to the figures, FIG. 4 illustrates an SAO method, according toone embodiment. As is typical with SAO imaging, selective excitation (orillumination) 104 is applied to an imaging target 102, and the lightscattered or fluoresced from the imaging target 102 is captured byoptical imaging 106. Here, the imaging target 102 is assumed to be a DNAmicroparticle such as those illustrated in FIGS. 2A-2C, 3A, and 3B, mRNA(messenger RNA) segments, lncRNA (long no-coding RNA), or proteins thatappear as spots or particles when fluorescently labeled and imaged suchas those illustrated in FIGS. 10-11 , but is not limited hereto. As willbe explained in more detail below with reference to FIGS. 7A-7D,selective excitation 104 is applied to the imaging target 102 by anillumination apparatus that is configured to cause interference of twolight beams on the imaging target 102. The excited target 102 emitssignals (or photons), and the emitted signals are captured in an opticalimaging system 106 including an objective lens and an imaging sensor (orimager). Then, it is determined 408 whether the images corresponding toall M phases of the 2D sinusoid excitation pattern were obtained. Ifimages corresponding to all the phases of the 2D sinusoid excitationpattern were not obtained in step 408, the excitation phase is changed402 and steps 104, 106, 408 are repeated for the changed excitationphase. If images corresponding to all the phases of the 2D sinusoidexcitation pattern were obtained in step 408, then it is determined 410whether the images corresponding to all the 2D sinusoid selectiveexcitation patterns were obtained. If images corresponding to all the 2Dsinusoid excitation patterns were not obtained in step 410, theexcitation pattern is changed by using a different spatial frequency(e.g., changing the pitch 143 and the orientation ϕ of the 2D sinusoidpattern) and steps 104, 106, 408, 402, 410, 404 are repeated for thenext selective excitation pattern. Then, if images corresponding to allthe 2D sinusoid excitation patterns were obtained in step 410, then thecaptured images are sent to a computer for SAO post processing 412 andvisualization to obtain the high-resolution images 114 of the imagingtarget 102 from the captured lower resolution raw images. As explainedabove, the raw images captured by optical imaging 106 have a resolutioninsufficient to resolve the objects on the imaging target 102, while thehigh resolution image 114 reconstructed by SAO post-processing 412 havea resolution sufficient to resolve the objects on the imaging target102.

The SAO method of the present invention uses an optimized number N ofselective excitation patterns and an optimized number M of excitationphases of each selective excitation pattern, so that SAO can be used toimage targets such as DNA microparticles in a massively parallel mannerwithin a short amount of time. As explained above, the number ofselective excitation patterns used in conventional SAO is determinedmerely by the hardware characteristics of the illumination system,independent and without consideration of the imaging target or theimaging system (objective lens and camera). Thus, the number of k-spacesampling points corresponding to the selective excitation patterns inconventional SAO was not optimized, and has many redundant and sometimesirrelevant k-space sampling points. In contrast, SAO according to theembodiments of the present invention herein uses selective excitationpatterns whose number N is optimized and minimized as a function of theimaging target's physical characteristics corresponding to spatialfrequency content (e.g., the size, shape, and/or spacing of the objectson the imaging target). SAO according to the embodiments herein may alsouse selective excitation patterns whose number N is optimizedalternatively or additionally as a function of various parameters of theimaging system (e.g., magnification (Mag) of the objective lens,numerical aperture (NA) of the objective lens, wavelength λ_(E) of thelight emitted from the imaging target, and/or effective pixel size p ofthe pixel sensitive area of the CCD, etc.). In this manner, theresulting number N of excitation patterns used in SAO becomes muchsmaller than that in conventional SAO, thereby enabling SAO to be usedwith DNA sequencing that requires massive parallelization of SAO imagingin a short amount of time to make DNA sequencing commercially feasible.Thus, dramatic reduction of cost and increase of throughput of DNAsequencing can be achieved.

FIG. 5A illustrates the k-space sampling points (selective excitationpatterns) used in SAO, according to one embodiment. In FIG. 5A, it isassumed that the CCD imaging area has a square shape and thus a squareshaped k-space sampling space 500 for SAO is also assumed, although thedescription for FIG. 5A below can be applied to a non-square shaped(e.g., rectangular) k-space sampling space as well. The k-space samplingspace 500 has an area of FOV², with the extent of the k-space samplingspace in each of the horizontal and vertical directions being FOV. Here,FOV stands for the k-space field of view. In the k-space frequencydomain, FOV should be equal to (1/Δx), where Δx the spatial resolutionof an imaging system (i.e., Δx is the minimum distance of two pointobjects that can be resolved by the imaging system). Each conjugate pair502, 506 and its DC point 504 correspond to one selective excitationpattern for SAO as used with the present invention. Thus, the number ofselective excitation patterns used in SAO corresponds to the number ofconjugate pairs of k-space points in the k-space sampling space 500(FOV×FOV). Δk_(x) is the k-space sampling interval, and is equal to(1/PFOV) where PFOV is the pixel field of view. The smaller the k-spacesampling interval Δk_(x) is in the k-space sampling space 500, thelarger the number of k-space points and the corresponding number ofexcitation patterns are. Specifically, the following equations hold:N=floor(L/2)  (Equation 1),

where L is the number of k-space points in the k-space, N is the numberof selective excitation patterns, and floor ( ) rounds the number to thenearest but smallest integer;L=round((FOV/Δk _(x))²)=round((PFOV/Δx)²)  (Equation 2),

where round ( ) rounds the number to the nearest integer, and PFOV isthe extent in the reciprocal (or Fourier) space of the sampling space(k-space) to be reconstructed from the samples.

FIG. 5B illustrates the selection of the k-space sampling interval usedin SAO, according to one embodiment. As explained above, the imagingtarget size determines the required spatial resolution Δx. Magnification(Mag) and CCD pixel size (Z) determines the effective pixel size p onthe imaging-target plane, p=Z/Mag. As shown in FIG. 5B, the detectedarea w(x) (i.e., the area captured by the pixel) can be represented asthe convolution of the pixel-sensitivity function p(x) (e.g., therectangular function with width p) and the point-spread function (PSF)h(x) of the lens (e.g., a bell-shaped curve). The width w can be definedas the 1/e² width of detected area w(x). Since the PSF of the lens isdetermined by the NA of the lens, the extent of the detected area (w)and the weighting over the detected area (i.e., the effectivesensitivity profile over the detected area) are the function of themagnification (Mag) of the lens, numerical aperture (NA) of the lens,and the CCD pixel size (Z).

As can be seen from the above, the k-space sampling space (FOV) isdetermined by the desired spatial resolution Δx and is dictated by theimaging target. The particles of interest, such as biological particleslike DNA microparticles or mRNA segments typically have a very smallsize, resulting in a large k-space sampling space. In conventional SAO,the k-space sampling interval Δk_(x) is set without regard to thephysical characteristics of the imaging target or the parameters of theimaging system, and is rather just set randomly according to whateverinterval allowed by the SAO illumination system. This made the number ofk-space points and the resulting selective excitation patternsprohibitively large for use in DNA sequencing applications using SAO,because of the high cost and low throughput of DNA sequencing using suchlarge number of selective excitation patterns in SAO.

In contrast, SAO according to the embodiments of the present inventionherein use selective excitation patterns whose number N is optimized asa function of the imaging target's physical characteristicscorresponding to spatial frequency content (e.g., the size, shape,and/or spacing of the imaging target). As shown in FIG. 5B, in oneembodiment, the pixel field of view PFOV is selected to be smaller thanthe extent (w) of the detected area, i.e., PFOV<w. Using a small PFOVresults in a larger k-space sampling interval Δk_(x), thereby reducingthe number (L) of k-space points in the k-space sampling space 500 andthe resulting number (N) of selective excitation patterns for use inSAO. As will be explained in more detail below with reference to FIGS.6A and 6B, using PFOV smaller than the extent (w) of the detected areacauses aliasing in the high resolution image obtained from SAO, but suchaliasing can be removed using the method as described below withreference to FIG. 6C. In other embodiments, the PFOV may be set to beequal to or larger than the extent (w) of the detected area, therebypreventing aliasing from occurring in the high resolution image obtainedfrom SAO. Also note that setting PFOV with consideration of the extent(w) of the detected area effectively sets the k-space sampling interval(Δk_(x)) and the resulting number (N) of selective excitation patternsbased on the various parameters of the imaging system, since the extent(w) of the detected area is a function of the magnification (Mag) of thelens, numerical aperture (NA) of the lens, and the CCD pixel size (Z) asexplained above.

Furthermore, SAO according to the embodiments herein further reduces thenumber of iterations of selective excitation and imaging by minimizingthe number of phase changes (M in steps 402, 408 of FIG. 4 ). Referringback to FIG. 5A and as explained above, one conjugate pair 502, 506 ofk-space points corresponds to one SAO interference pattern generationmodule that produces a specific pitch and orientation of one selectiveexcitation pattern. The DC point 504 corresponds to the signal offset ofthe 2D sinusoid selective excitation pattern. Thus, in one embodiment,three different measurements at three different phases of theinterference pattern with the same pitch and orientation are made todistinguish between the two conjugate points 502, 506 and the DC point504 in the k-space. This is in contrast to conventional SAO, where morethan three phases were used to illuminate and image each selectiveexcitation pattern for SAO. In another embodiment, since the DC point504 is common for all conjugate pairs 502, 506, it is also possible toutilize the DC point 504 obtained in one 2D sinusoid pattern with aspecific pitch and orientation to obviate the need for illuminating andimaging the selective excitation pattern at the DC point 504 of anotherselective excitation pattern with a different pitch and orientation,thereby reducing the number M of changed phases needed for imaging insteps 402, 408 (FIG. 4 ) to two (2) phases for the other selectiveexcitation patterns. In other words, each interference patterngeneration module produces a pattern with only two different phases,except one module that produces pattern with three different phases toacquire the DC point 504. For optimal tolerance to noise, one can choosespecific phases for the patterns. For three different phases per aspecific selective excitation pattern, the optimal phase difference maybe 0, 120, and 240 degrees. For two different phases per a specificselective excitation pattern, the optimal phase difference may be 0 and90 degrees.

Since the objects of interest (i.e., biological particles such as DNAmicroparticles, mRNA segments, lncRNA, or proteins) are typicallycircularly symmetric, the k-space spectrum of the objects of interestwill also be circularly symmetric and thus only k-space samples in thecircular region with diameter of FOV (=1/Δx) may be needed for SAO.Thus, in one embodiment, the SAO according to the present invention usesselective excitation patterns corresponding to the k-space samplingpoints within the circular region 512, as shown in FIG. 5C.

FIG. 5D illustrates reducing the number of k-space sampling points bysparse k-space sampling, according to one embodiment. Conventional SAOmethods do not utilize frequency information of the objects in the imagescene. Solid objects such as beads used in microparticles have much lessenergy in the high spatial-frequencies compared to the low frequencies.Therefore, under-sampling in the high spatial frequencies is moretolerable than under-sampling in the low spatial frequency region. Thus,in one embodiment of the present invention, the number (N) of selectiveexcitation patterns is further reduced by non-uniform orvariable-density sampling in the Fourier space as shown in FIG. 5D. Thepenalty for not meeting the Nyquist sampling rate in high spatialfrequencies is tolerable in SAO for DNA sequencing applications, andthus SAO according to the embodiments herein relaxes the Nyquistsampling criteria in the higher-frequencies, thereby reducing the numberof selective excitation patterns by almost half of what would berequired with uniform sampling. For example, the number of k-spacesamples in the embodiment of FIG. 5D is only 54% of the number ofk-space samples in the embodiment of FIG. 5C.

FIG. 6A illustrates how aliasing occurs in SAO by use of a pixel fieldof view smaller than the detected area, according to one embodiment. Asmentioned above with reference to FIG. 5B, using a pixel field of view(PFOV) smaller than the extent (w) of the detected area, i.e., PFOV<w,results in aliasing in the image obtained for SAO, because each pixel inthe CCD would detect areas larger than the pixel itself. The extra area(i.e., left and right parts of the extent (w) outside of p(x)) is thearea also detected by its neighboring pixels in the CCD. This isillustrated in FIG. 6A, where the objects 602, 604, 606, 608 detected inthe extra area in the neighboring pixels will enter into the centerpixel 610 (assuming rectilinear sampling in the k-space), resulting inaliasing and unwanted artifacts that degrade the image quality.

FIG. 6B illustrates how the actual signal at a pixel of an imagingsystem may be determined by unfolding the measured signal at the pixelto remove aliasing, according to one embodiment. In order to removealiasing in the measured image signal and obtain the actual imagesignals, one can formulate a linear equation in the form of y=Ax at aparticular sub-pixel k at pixel CCD_(i) of the CCD. Referring to FIG.6B, m_(k,i) represents the measured signal (including aliasing) at aparticular k-th sub-pixel location within the i-th CCD pixel CCD_(i).Note that the relative locations of all measured signals m_(k,i) (i=1, .. . , ∞) within their i-th CCD pixels are the same. s_(k,i) representsthe actual or ideal signal of the object at the k-th sub-pixel locationswithin the i-th CCD pixel CCD_(i). α, β, and γ represent the values ofthe weighting function w(x) of the i-th CCD pixel at the locationscorresponding to s_(k,i−1), s_(k,i), and s_(k,i+1), respectively, ands_(k,i) (i=1, . . . , ∞) is the actual (ideal) signal at the particulark-th sub-pixel location within the i-th CCD pixel CCD_(i). As explainedabove, the weighting function w(x) can be represented as the convolutionof the pixel-sensitivity function p(x) (e.g., the rectangular functionwith width p) and the point-spread function (PSF) h(x) of the lens(e.g., a bell-shaped curve). With these parameters defined and assumingthat the number of pixels of the CCD is infinite, one can write thesignal-equation series for a particular k^(th) sub-pixel location as alinear matrix equation y=Ax, where y=[m_(k,1), m_(k,2), . . . ],x=[s_(k,1), s_(k,2), . . . ], and A is a matrix with elements beingzeros and values of the weighting function (e.g., α, β, and γ). Thelinear matrix equation y=Ax shows that the “unfolding” process (i.e.,recovering the actual signal s_(k,i)) can be viewed as a common inverseproblem of y=Ax (i.e., x=A⁻¹y). In other embodiments, if non-rectilinearsampling pattern is used (e.g., variable-density, radial sampling,etc.), the actual relationship between s_(i) and m_(i) will change fromthat shown in FIG. 6B, in which case the point-spread-function (i.e.,impulse response) can be measured in either simulation or realexperiments to construct the inversion matrix (A⁻¹).

FIG. 6C illustrates a method of unfolding the measured signal at thepixel to remove aliasing, according to one embodiment. The steps 652,654, 656 together constitute the post-processing steps for SAO. Inconventional SAO, post-processing includes only the regular SAOreconstruction 652 to generate the high spatial resolution image 653from the low resolution images (M×N) 650 obtained from selectiveexcitation of the imaging target. However, in SAO according to theembodiments of the present invention herein, post-processing includesthe “unfolding” step 670 to remove aliasing from the high spatialresolution image 653 that contains aliasing resulting from using PFOVsmaller than the extent (w) of the detected area for selectiveexcitation. The unfolding process 670 includes solving the linearequation y=Ax to recover the actual signals x, which is repeated 656 ateach sub-pixel location, for all CCD pixels. As a result, a high spatialresolution image 658 without aliasing can be obtained from SAO, despiteusing PFOV smaller than the extent (w) of the detected area forselective excitation in SAO.

Note that “unfolding” as explained herein can also be used to improvethe SAO image reconstruction quality even when PFOV larger than or equalto the extent (w) of the detected area is used for selective excitationin SAO. In conventional SAO, the reconstructed pixels are simply cropped(to the width being p) and stitched together. This way of “crop andstitch” still does not undo the apodization caused by the weightingfunction w(x). In contrast, “unfolding” may be used according to thepresent invention even when PFOV larger than or equal to the extent (w)of the detected area is used for selective excitation in SAO such thatno aliasing occurs. Since the unfold process is fundamentally undoing(i.e., unapodizing) the weighting function w(x), the “unfold” processcan also be used to improve image reconstruction even when PFOV>=w isused for SAO selective excitation.

FIG. 7A illustrates a structured illumination apparatus for selectivelyexciting the microparticles, according to one embodiment. Theillumination apparatus shown in FIG. 7A is merely exemplary, and variousmodifications may be made to the configuration of the illuminationapparatus for SAO according to the present invention. The exampleillumination apparatus in FIG. 7A shows only two interference patterngeneration modules (IPGM) 712, 713 for simplicity of illustration, butfor biological sequencing applications such as real DNA sequencingapplications, mRNA sequencing applications, or lncRNA sequencingapplications there would be a larger number of IPGMs. Each IPGM is inmodular form and is configured to generate one selective excitationpattern at a given pitch and orientation, corresponding to one conjugatepair of the k-space sampling points. Thus, there is a one-to-onerelationship between an IPGM and a 2-D sinusoid selective excitationpattern at a given pitch and orientation and to one conjugate pair ofthe k-space sampling points. A larger number (N) of selective excitationpatterns would require a larger number of IPGMs in the SAO illuminationapparatus.

The structured illumination apparatus 700 generates multiplemutually-coherent laser beams, the interference of which producesinterference patterns. Such interference patterns are projected onto themicroparticle array substrate 204 and selectively excite the DNAmicroparticles 202. Using the interference of multiple laser beams togenerate the interference patterns is advantageous for many reasons. Forexample, this enables high-resolution excitation patterns with extremelylarge FOV and DOF. Although the structured illumination apparatus ofFIG. 7A is described herein with the example of generating excitationpatterns for DNA microparticles, it should be noted that the structuredillumination apparatus of FIG. 7A can be used for any other type ofapplication to generate excitation patterns for imaging any other typeof target, such as biological particles including DNA segments, mRNA(messenger RNA) segments, lncRNA (long no-coding RNA), or proteins thatappear as spots or particles when fluorescently labeled and imaged.Examples of imaging mRNA segments using selective excitation patternsare described below in conjunction with FIGS. 10-11 .

Referring to FIG. 7A, the structured illumination apparatus 700 includesa laser 702, a beam splitter 704, shutters 705, 707, fiber couplers 708,709, a pair of optical fibers 710, 711, and a pair of interferencepattern generation modules (IPGMs) 712, 713. As explained above, eachIPGM 712, 713 generates an interference pattern (selective excitationpattern) that corresponds to one conjugate pair of k-space samplingpoints. The beam 703 of the laser 702 is split by the beam splitter 704into two beams 740, 742. A pair of high-speed shutters 705, 707 is usedto switch each beam 740, 742 “on” or “off” respectively, or to modulatethe amplitude of each beam 740, 742, respectively. Such switched laserbeams are coupled into a pair of polarization-maintaining optical fibers711, 710 via fiber couplers 709, 708. Each fiber 711, 710 is connectedto a corresponding interference pattern generation module 713, 712,respectively. The interference pattern generation module 713 includes acollimating lens 714′, a beam splitter 716′, and a translating mirror718′, and likewise the interference pattern generation module 712includes a collimating lens 714, a beam splitter 716, and a translatingmirror 718.

The beam 744 from the optical fiber 710 is collimated by the collimatinglens 714 and split into two beams 724, 726 by the beam splitter 716. Themirror 718 is translated by an actuator 720 to vary the opticalpath-length of the beam 726. Thus, an interference pattern 722 isgenerated on the substrate 204 in the region of overlap between the twolaser beams 724, 726, with the phase of the pattern changed by varyingthe optical path-length of one of the beams 726 (i.e., by modulating theoptical phase of the beam 726 by use of the translating mirror 718).

Similarly, the beam 746 from the optical fiber 711 is collimated by thecollimating lens 714′ and split into two beams 728, 730 by the beamsplitter 716′. The mirror 718′ is translated by an actuator 720′ to varythe optical path-length of the beam 728. Thus, the interference pattern722 is generated on the substrate 204 in the region of overlap betweenthe two laser beams 728, 730, with the pattern changed by varying theoptical path-length of one of the beams 728 (i.e., by modulating theoptical phase of the beam 728 by use of the translating mirror 718′).

As shown in FIG. 7A, each IPGM 712, 713 is implemented in modular formaccording to the embodiments herein, and one IPGM produces aninterference pattern corresponding to one conjugate pair of k-spacepoints. This modularized one-to-one relationship between the IPGM andthe k-space points greatly simplifies the hardware design process forSAO according to the embodiments herein. As the number of selectiveexcitation patterns used for SAO is increased or decreased, the SAOhardware is simply changed by increasing or decreasing the number ofIPGMs in a modular manner. In contrast, conventional SAO apparatuses didnot have discrete interference pattern generation modules but had aseries of split beams producing as many multiple interferences aspossible. Such conventional way of designing SAO apparatuses producednon-optimized or redundant patterns, slowing down and complicating theoperation of the SAO system.

While this implementation illustrated in FIG. 7A is used for itssimplicity, various other approaches can be used within the scope of thepresent invention. For example, the amplitude, polarization, direction,and wavelength, in addition to or instead of the optical amplitude andphase, of one or more of the beams 724, 726, 728, 730 can be modulatedto change the excitation pattern 722. Also, the structured illuminationcan be simply translated with respect to the microparticle array tochange the excitation pattern. Similarly, the microparticle array can betranslated with respect to the structured illumination to change theexcitation pattern. Also, various types of optical modulators can beused in addition to or instead of the translating mirrors 718, 718′,such as acousto-optic modulators, electro-optic modulators, a rotatingwindow modulated by a galvanometer and micro-electro-mechanical systems(MEMS) modulators. In addition, although the structured illuminationapparatus of FIG. 7A is described herein as using a laser 702 as theillumination source for coherent electro-magnetic radiation, other typesof coherent electro-magnetic radiation sources such as an SLD(super-luminescent diode) may be used in place of the laser 702.

Also, although FIG. 7A illustrates use of four beams 724, 726, 728, 730to generate the interference pattern 722, larger number of laser beamscan be used by splitting the source laser beam into more than two beams.For example, 64 beams may be used to generate the interference pattern722. In addition, the beam combinations do not need to be restricted topair-wise combinations. For example, three beams 724, 726, 728, or threebeams 724, 726, 730, or three beams 724, 728, 730, or three beams 726,729, 730, or all four beams 724, 726, 728, 730 can be used to generatethe interference pattern 722. Typically, a minimal set of beamcombinations (two beams) is chosen as necessary to maximize speed. Also,the beams can be collimated, converging, or diverging. Althoughdifferent from the specific implementations of FIG. 7A and for differentapplications, additional general background information on generatinginterference patterns using multiple beam pairs can be found in (i) U.S.Pat. No. 6,016,196, issued on Jan. 18, 2000 to Mermelstein, entitled“Multiple Beam Pair Optical Imaging,” (ii) U.S. Pat. No. 6,140,660,issued on Oct. 31, 2000 to Mermelstein, entitled “Optical SyntheticAperture Array,” and (iii) U.S. Pat. No. 6,548,820, issued on Apr. 15,2003 to Mermelstein, entitled “Optical Synthetic Aperture Array,” all ofwhich are incorporated by reference herein.

FIG. 7B illustrates the arrangement of the illumination patterngeneration modules in a half ring structure according to one embodiment.Referring to FIG. 7B, multiple IPGMs (IPGM 1, IPGM 2, . . . , IPGM N)such as IPGMs 712, 713 (FIG. 7A) are arranged substantiallysymmetrically in a half-ring shape on a half-ring shaped, monolithicstructure 762, to generate the selective excitation patterns. Thehalf-ring structure 762 is fixed on the system table 768. In theembodiment of FIG. 7B, the N IPGMs generate N selective excitationpatterns for SAO on the imaging target 102, and the scattered orfluoresced light 752 is passed through objective lens 124 and captured756 by camera 126 which may be a CCD camera.

These arrangements of the IPGMs in the embodiment of FIG. 7B enable amonolithic and compact holding structure that has multiple benefits forenabling the SAO system to be used for DNA sequencing applications,compared to conventional optical-bench SAO systems where each opticalcomponent is individually mounted on its holding structure. Themonolithic structure 762 enables the IPGM arrangement to be compact andsymmetric, and this compact, symmetric, and monolithic structurepreserves more stable channel-to-channel and beam-to-beam geometryagainst mechanical and thermal distortions. The compact monolithicstructure 762 is also less susceptible to non-flatness or torsional andbending modes of the optical table 768, and the symmetric arrangement ofthe IPGMs around the half-ring structure 762 makes the effect of heatcontraction or expansion less detrimental to the beam geometry, i.e.,the channel-to-channel or beam-to-beam angles of laser beams are changedless compared to a non-symmetric structure. Furthermore, the compactdesign shortens the travel distances of the laser beam in air, making iteasy to prevent air disturbances affecting the stability of theinterference pattern that may cause the effective optical path length tochange resulting in change of the interference fringe position. Suchstability allows more accurate calibration of the beam geometry.Furthermore, the half-ring arrangement of the IPGMs in FIG. 7B has theadditional advantage that it enables the imaging module (i.e., camera126 and objective lens 124), illumination structure (i.e., the half-ring762), and the imaging target 102 to be placed on one stiff structure(e.g., optical table) 768.

FIG. 7C illustrates the internal structure of an illumination patterngeneration module, according to one embodiment. The embodiment of FIG.7C has a rotating window 760 in IPGM 750 that is placed after the mirror762. The beam 770 from the optical fiber 710 is collimated by thecollimating lens 754 and the collimated beam 744 is split into two beams773, 774 by the beam splitter 756. Beam 773 is reflected by mirror 758and the reflected beam 778 is projected onto the imaging target togenerate the interference pattern 780. Beam 774 is reflected by mirror762 and the optical path-length of the reflected beam 776 is modulatedby optical window 760 that is rotated, using a galvanometer, therebymodulating the optical phase of the corresponding beam 776 andgenerating a modulated beam 777. The interference pattern 780 isgenerated in the region of overlap between the two laser beams 777, 778,with the pattern changed by varying the optical path-length of one ofthe beams 777. By placing the rotating window 760 after the mirror 762,the width W_(IPGM) and the size of IPGM 750 can be reduced, as comparedto the embodiment of FIG. 7A and FIG. 7D illustrated below. Thus, thehalf-ring shaped structure 762 holding the IPGMs can be made morecompact, since the width W_(IPGM) of the IPGM directly affects theradius of the half-ring, for example, as shown in FIG. 7B.

FIG. 7D illustrates the internal structure of an illumination patterngeneration module, according to another embodiment. IPGMs in theembodiments of FIGS. 7A and 7C may produce two beams that do not haveequal path length between the interfering point at the imaging targetand the splitting point (i.e., the beam splitter). The non-equal pathlength may significantly reduce the sinusoidal contrast if a relativelyshort coherent-length laser is used and also limit the applicability ofthe SAO system to only a specific wavelength (e.g., 532 nm green laser)since only a small number of lasers with specific wavelengths have asufficiently long coherent-length that can be used with suchnon-equal-path IPGMs for good sinusoidal contrast. Compared to theembodiment of FIG. 7A, the embodiment of FIG. 7D uses additional foldingmirrors to achieve equal paths between the two split beams. The laserbeam 744 is split into beams 781, 780 by beam splitter 756. Beam 781 isreflected by mirror 782 and its optical path-length is modulated byrotating window 760 to generate beam 788. On the other hand, beam 780 isreflected twice by two mirrors 784, 787 to generate the reflected beam789. Beam 788 and 789 eventually interfere at the imaging target togenerate the selective excitation patterns. By use of two mirrors 784,786, the optical path 744-780-785-789 is configured to have a lengthsubstantially equal to the length of the optical path 781-783-788. Thisequal-path scheme allows lasers with short coherent lengths to be usedto generate interference patterns with high contrast. Moreover, thisequal-path scheme enables the SAO system to be used with wavelengthsother than 532 nm, thus making multiple-color SAO practical.

Detection of Particles on Target Using Structured Illumination Patterns

Conventionally, a detection system estimates whether particles arepresent on a plurality of regions on a target by generating a set ofreconstruction estimates based on the intensity values of thereconstructed image. For example, a reconstruction estimate for a pixellocation of the reconstructed image may indicate whether a particle ispresent at a corresponding region of the target 102 by comparing theintensity value for the pixel location to a predetermined threshold.However, it is often difficult to detect particles with high accuracy inthis manner due to, for example, the texture of the target that resultsin a noisy reconstruction image.

The particle detection method according to various embodiments of thepresent disclosure detects presence and location of particles on atarget using measured signals from a plurality of structuredillumination patterns. Specifically, the particle detection methoddisclosed herein uses measured signals obtained by illuminating thetarget with structured illumination patterns to detect particles. Aparticle may respond differently to illumination across multiplestructured illumination patterns, and the degree of variation in thesemeasured signals in raw images can provide significant insight fordetermining whether a particle is present on the target 102. While thereconstruction process generates a reconstructed image at a higherresolution than the raw images, the intensity values of thereconstructed image do not typically preserve this degree of variationthat is useful for particle detection.

Turning to the figures, FIG. 8 illustrates a particle detection method,according to one embodiment. The imaging target 102 is illuminated 806with a plurality of structured illumination patterns that are eachcharacterized by a set of illumination characteristics. In oneembodiment, the structured illumination patterns are the selectiveexcitation patterns described herein that are characterized by at leastspatial frequency and phase. For example, the image target 102 may beilluminated with three structured illumination patterns that have a setof illumination characteristics {Spatial Frequency 1, Phase 1}, {SpatialFrequency 1, Phase 2}, and {Spatial Frequency 1, Phase 3}. The imagingtarget 102 may be assumed to be various biological molecules of interestsuch as DNA segments, mRNA (messenger RNA) segments, lncRNA (longno-coding RNA), or proteins that appear as spots or particles whenfluorescently labeled and imaged, but is not limited hereto. The phasemay be generated by varying the optical path-length of one of the laserbeams used to generate the pattern, for example, by modulating theoptical phase of the beam by use of a translating mirror. A plurality ofraw images of the image target are generated 808 by measuring opticalsignals from the illuminated target 102. Each raw image may be obtainedby illuminating the target 102 with a particular structured illuminationpattern, and includes raw intensity values arranged as a set of pixelsfor the raw image. For example, a raw image may be generated when theimage target 102 is illuminated with each of the three structuredillumination patterns, resulting in three raw images Raw Image 1, RawImage 2, and Raw Image 3. The raw intensity values may be captured bythe optical imaging system 106 when the excited target 102 emits signals(or photons). Each pixel in a raw image may correspond to a particularregion in the target 102, and the intensity value for the pixel locationis obtained by measuring signals emitted from the particular region ofthe target 102 when the target 102 was illuminated with a structuredillumination pattern.

For each region in one or more regions of the image target 102, amodulation estimate using the raw images are generated that indicateswhether a particle is present at the region of the target 102.Specifically, a modulation score is determined 810 for each region ofinterest by combining a set of raw intensity values that correspond tothe region of interest in the target 102. The set of raw intensityvalues are obtained from the plurality of raw images that were generatedby imaging the target 102 with the plurality of structured illuminationpatterns. A particular region of the target 102 may have a correspondingpixel location in each raw image, and the modulation score for thatparticular region can be generated by combining each raw intensity valuefrom the corresponding raw image together. For example, raw intensityvalues for a pixel location in Raw Image 1, Raw Image 2, and Raw Image 3that correspond to the particular region of the target 102 may becombined to generate one modulation score for that region of the target102. Alternatively, a modulation score can be determined for a region ofthe target 102 that encompasses more than a single pixel location. Inthis case, measurements from the region of the target 102 may have agroup of corresponding pixel locations in each raw image, and themodulation score for that region can be generated by combining eachgroup of raw intensity values from the corresponding raw image together.

The modulation score indicates a degree of variation in the set of rawintensity values, and predicts a likelihood that a particle is presenton the region of the target 102 based on the observed raw intensityvalues. The modulation score is compared 812 to a first predeterminedthreshold to generate a modulation estimate indicating presence of theparticle on the region of the target 102. Thus, each pixel location or agroup of pixel locations in the raw images may be labeled with amodulation estimate indicating whether a particle or a part of aparticle is present at the particular region of the target 102corresponding to that pixel location. In one embodiment, a positivemodulation estimate indicates that a particle is present on the target102 if the modulation score is equal to or above the predeterminedthreshold. In general, a high modulation score indicates a higherlikelihood of presence of a particle for that pixel. Typically, if aparticle is not present in the corresponding location, the variation inpixel intensities according to changes in illumination characteristicssuch as phase changes, would be relatively small or constant. Thus, ahigh degree of variation indicates the presence of a particle in thelocation of the pixel.

Given a series of a set of raw intensity values for a particular regionof the target 102, the modulation score indicates a degree of variationin the set of raw intensity values. In one instance, the modulationscore is determined as the standard deviation between the set of rawintensity values. In another instance, the modulation score isdetermined as the standard deviation between the set of raw intensityvalues normalized (divided) by the mean of the set of raw intensityvalues. In another instance, the modulation score is determined as therange of the set of raw intensity values that indicates the differencebetween the maximum value and the minimum value of the set. In yetanother instance, the modulation score is determined as agoodness-of-fit metric that indicates how well the set of raw intensityvalues fit to an expected curve when the particle is present at theregion of the image target 102. For example, the goodness-of-fit mayindicate how well the set of raw intensity values fir a sine curve ofillumination phase vs. intensity. Moreover, it is appreciated that themodulation score may be generated by any transformation of these metricsas well, for example, scaling these metrics by a constant factor,addition or subtraction of certain terms, and the like.

In one embodiment, the modulation score for a particular region of thetarget 102 can also be generated by determining one or moresub-modulation scores from one or more subsets of raw images, andcombining the sub-modulation scores to determine the modulation scorefor the region of the target 102, as will be described in more detailbelow in conjunction with Table 1.

Table 1 illustrates an example of determining modulation scores for aset of structured illumination patterns including K Spatial Frequenciesand M Phases (therefore, N=K×M). In particular, Table 1 shows the caseof K=4 Spatial Frequencies (Spatial Frequency 1, Spatial Frequency 2,Spatial Frequency 3, and Spatial Frequency 4) and M=3 Phases (Phase 1,Phase 2, and Phase 3), for a total of 12 structured illuminationpatterns. While the example in Table 1 uses the same number of phasesfor all spatial frequencies, this is merely an example, and differentnumber of phases can be used for each different spatial frequency inother instances.

In Table 1, modulation scores for a particular region of interest of thetarget 102 that corresponds to a single pixel location in each raw imageare determined using raw intensity values from these pixel locations.Thus, the particular region of the target 102 is associated with 12 rawpixel intensity values, each obtained from a raw image generated byilluminating the target 102 with a corresponding structured illuminationpattern. In this example, four modulation scores, MS1, MS2, MS3, andMS4, are determined for this particular region of the target 102. Eachmodulation score is determined by combining the raw pixel intensityvalues that are obtained from a subset of raw images with same spatialfrequency. For example, “IS1” represents raw pixel intensity values forthe region of the image target 102 obtained by illuminating the imagetarget 102 with structured illumination patterns {Spatial Frequency 1,Phase 1}, {Spatial Frequency 1, Phase 2}, {Spatial Frequency 1, Phase3}, “IS2” represents raw pixel intensity values obtained by illuminatingthe image target 102 with structured illumination patterns {SpatialFrequency 2, Phase 1}, {Spatial Frequency 2, Phase 2}, {SpatialFrequency 2, Phase 3}, “IS3” represents raw pixel intensity valuesobtained by illuminating the image target 102 with structuredillumination patterns {Spatial Frequency 3, Phase 1}, {Spatial Frequency3, Phase 2}, {Spatial Frequency 3, Phase 3}, and “IS4” represents rawpixel intensity values obtained by illuminating the image target 102with structured illumination patterns {Spatial Frequency 4, Phase 1},{Spatial Frequency 4, Phase 2}, {Spatial Frequency 4, Phase 3}. “MS1”represents the modulation score for subset IS1, “MS2” represents themodulation score for subset IS2, “MS3” represents the modulation scorefor subset IS3, and “MS4” represents the modulation score for subsetIS4. As described above, each modulation score may be determined by oneor a combination of taking the standard deviation, normalizing thestandard deviation by the mean, the range, and a goodness-of-fit metricfor the corresponding subset of raw intensity values, among other waysto determine degree of variation. In addition, the modulation scores fora particular region of the target 102 may also be generated by combininga group of pixel intensity values from each raw image.

In one instance, modulation scores for individual subsets can be used togenerate the modulation estimate of whether a particle is present at theparticular region of the image target 102. For example, a detectionsystem may use only modulation score MS1 to determine the presence of aparticle by comparing the score to a predetermined threshold. In thisexample, the modulation estimate may indicate a particle is present ifmodulation score MS1 is equal to or above a threshold, or may indicate aparticle is not present if MS1 is below the threshold. In anotherinstance, two or more of the scores MS1, MS2, MS3, and MS4 can beconsidered as sub-modulation scores, and these sub-modulation scores arecombined to generate a final modulation score for the region of theimage target 102. For example, the final modulation score “MS” can bedetermined as the multiplication of all four sub-modulation scoresMS1×MS2×MS3×MS4, and the detection system may use the final score MS todetermine the presence of a particle by comparing the final score to apredetermined threshold. Similarly, the modulation estimate may indicatea particle is present if the final modulation score MS is equal to orabove a threshold, or may indicate a particle is not present if MS isbelow the threshold. Although multiplication is used as an example,other operations may be used in other embodiments to combinesub-modulation scores, such as addition, multiplication, and the like.

TABLE 1 Spatial Frequency (K = 4) 1 2 3 4 Position 1 IS1 IS2 IS3 IS4 (M= 3) 2 3 Modulation 1. MS1 = 1. MS2 = 1. MS3 = 1. MS4 = Score (MS)Standard Standard Standard Standard Deviation Deviation DeviationDeviation (IS1) (IS2) (IS3) (IS4) 2. MS1 = 2. MS2 = 2. MS3 = 2. MS4 =Max (IS1) − Max (IS2) − Max (IS3) − Max (IS4) − Min (IS1) Min (IS2) Min(IS3) Min (IS4) 3. MS1 = 3. MS2 = 3. MS3 = 3. MS4 = Goodness of Goodnessof Goodness of Goodness of fit between fit between fit between fitbetween IS1 and the IS2 and the IS3 and the IS4 and the structuredstructured structured structured illumination illumination illuminationillumination 4. MS1 = 4. MS2 = 4. MS3 = 4. MS4 = Standard StandardStandard Standard Deviation Deviation Deviation Deviation (IS1)/Mean(IS2)/Mean (IS3)/Mean (IS4)/Mean (IS1) (IS2) (IS3) (IS4) IS: Series ofIntensities of a pixel that represent measurements from a particularregion of interest on the target

Turning to the figures, FIG. 9 illustrates a particle detection method,according to another embodiment. In this embodiment, modulationestimates are used to validate reconstruction estimates obtained fromreconstructed images to generate a set of combined estimates forparticle detection on a plurality of regions of the target 102. Thereconstruction estimates are compared to modulation estimates, and areused to validate the reconstruction estimates. Specifically, whilereconstruction estimates may not have the optimal accuracy,reconstructed images can provide particle detection estimates at ahigher granularity of regions than raw images if they have a higherimage resolution compared to the plurality of raw images. For example, areconstructed image may have 2 times the resolution of the plurality ofraw images, and the reconstruction estimate for 1 pixel location in thereconstructed image may correspond to a smaller area of the target 102than the modulation estimate for 1 pixel location in the plurality ofraw images. By validating the reconstruction estimates with modulationestimates of higher accuracy, the detection system can perform particledetection with both higher accuracy and with higher granularity.

Returning to FIG. 9 , modulation estimates are generated using aplurality of raw images through steps 906, 908, 910, and 912, which arelargely identical to steps 806, 808, 810, and 812 described inconjunction with FIG. 8 . A reconstructed image of the target 102 isgenerated by reconstructing 914 the plurality of raw images. In oneembodiment, when the plurality of raw images is generated by a pluralityof selective excitation patterns, the reconstruction process is the SAOreconstruction process or post-processing step described herein inconjunction with FIGS. 1A-1C and 6A-6C that generates a high spatialresolution reconstructed image. A set of reconstruction estimates aregenerated for pixel locations of the reconstructed image by comparing916 the intensity values of the reconstructed image to a predeterminedthreshold. For example, a reconstruction estimate may be generated foreach pixel location of the reconstructed image, and a positive estimatemay indicate a particle is present at a corresponding region of thetarget 102 if the intensity value for the pixel location is equal to orabove a threshold. The reconstruction estimate for a particular regionof the target 102 is compared 918 to the modulation estimate for theparticular region of the target 102 to generate a combined estimate forthe target 102. When the resolution of the reconstructed image is higherthan the resolution of the raw images, and the reconstructed imagecaptures the target 102 with higher granularity, a pixel in areconstructed image may correspond to an area on the target 102 that issmaller than an area imaged by a pixel in a raw image. In this instance,a reconstruction estimate for a pixel in the reconstructed image may becompared to a modulation estimate for a region of the target 102 thatencompasses or otherwise overlaps the region encompassed by thereconstructed pixel.

In one embodiment, the modulation estimates are used to decrease falsepositive errors for reconstruction estimates. A false positive erroroccurs when a particle is not present in a particular region of thetarget 102, but the reconstruction estimate indicates that a particle ispresent in that region. These errors may occur due to noisy backgroundimages in of the target 102 that include defects or other patterns thatappear to look like the particles of interest when in fact they are not.To decrease false positive errors, the detection system identifiesreconstructed pixels with positive reconstruction estimates, compares918 these estimates to the corresponding modulation estimates, andgenerates combined estimates as the final estimates for particledetection. A combined estimate for a reconstructed pixel with a positivereconstruction estimate indicates that a particle is present only if thecorresponding modulation estimate is positive, and indicates that aparticle is not present if the corresponding modulation estimate isnegative. In this manner, the modulation estimates can be used tovalidate positive reconstruction estimates in case they mistakenlydetect background patterns of the target 102 as the particles ofinterest.

In another embodiment, the modulation estimates can also be used todecrease false negative errors for reconstruction estimates. A falsenegative error occurs when a particle is present in a particular regionof the target 102, but the reconstruction estimate indicates that aparticle is not present in that region. To decrease false negativeerrors, the detection system identifies reconstructed pixels withnegative reconstruction estimates, compares these estimates to thecorresponding modulation estimates, and generates combined estimates asthe final estimates for particle detection. A combined estimate for areconstructed pixel with a negative reconstruction estimate indicatesthat a particle is not present only if the corresponding modulationestimate is negative, and indicates that a particle is present if thecorresponding modulation estimate is positive.

Methods and apparatus for obtaining images of the biological moleculesand samples are described in further detail in U.S. patent applicationSer. No. 15/059,245 filed on Mar. 2, 2016, now issued as U.S. Pat. No.9,772,505, which is incorporated by reference.

Examples of Particle Detection

FIG. 10 illustrates examples of particle detection methods that wereperformed on a tissue section target area, according to one embodiment.Part (a) of FIG. 10 illustrates an image of the tissue section obtainedby a conventional high resolution microscopy with oil immersion lens(100×/1.4 NA objective lens with z-stack). As shown in Part (a) of FIG.10 , the particles of interest (mRNA molecules) are shown as white spotsacross the tissue section. Part (b) of FIG. 10 illustrates areconstructed image of the target area obtained by performing a SAOreconstruction process on raw images. The raw images were obtained byilluminating the target area with a series of 12 distinct structuredillumination patterns formed by the interference of laser beams. Asshown in Part (b) of FIG. 10 , the particles appear as white spots onthe image. Part (c) of FIG. 10 illustrates the image of Part (b) of FIG.10 annotated (in “x” marks) with regions that are associated withpositive reconstruction estimates, specifically pixels that haveintensity values above a threshold, as described in conjunction with themethod of FIG. 9 . As shown in Part (c) of FIG. 10 , more annotationsappear than are particles on the target area, due to false positiveerrors. Part (d) of FIG. 10 illustrates the image of Part (b) of FIG. 10annotated (in “x” marks) with regions that are associated with positivecombined estimates, which are generated by comparing the annotatedregions in Part (c) of FIG. 10 with the corresponding modulationestimates, and retaining annotations for only those regions that havepositive modulation estimates, as described in conjunction with themethod of FIG. 9 . The modulation estimates were generated bydetermining 4 sub-modulation scores MS1, MS2, MS3, MS4 each respectivelyfor raw intensity values from a subset of raw images with same spatialfrequency as illustrated in Table 1, and determining the finalmodulation score as MS=MS1×MS2×MS3×MS4. Each sub-modulation score wasdetermined as the standard deviation normalized by the mean of thesubset of raw intensity values. As shown in Part (d) of FIG. 10 , thedetection system is able to detect the particles with improved accuracyand with a lower false positive error rate by validating thereconstruction estimates with modulation estimates.

FIG. 11 illustrates examples of particle detection methods that wereperformed on single molecule mRNA FISH (fluorescence in situhybridization) samples, according to one embodiment. Part (a) of FIG. 11illustrates a first annotated image of U2OS cells on a 96 well platelabeled for EGFR (Epidermal Growth Factor Receptor) mRNA by performingthe particle detection method of FIG. 9 to reduce false positive errorrates. Labeling of mRNA was done by Stellaris® mRNA FISH method usingprobes conjugated with Quasor® 670 dye, both of LGC BiosearchTechnologies. The image shown corresponds to one field of view (0.33mm×0.33 mm) of an SAO apparatus according to the embodiments describedherein. In this example, total of 146 cells and 3,492 mRNA spots weredetected from one field of view. Part (b) of FIG. 11 illustrates azoomed-in view of the image in Section (a) of FIG. 11 . Detected mRNA'sare shown as individual dots, and the line, as illustrated as an exampleas 1102A, shows the boundary of nucleus detected by nuclear segmentationsoftware. Part (c) of FIG. 11 illustrates a second annotated image of aregion from mouse brain tissue labeled for mRNA using probes conjugatedwith Cy5 dye. Name of the gene target and specific chemistry method forlabeling mRNA is not known. The image shown corresponds to one field ofview (0.33 mm×0.33 mm) of an SAO apparatus according to the embodimentsherein. In this example, total of 275 cells and 36,346 mRNA spots weredetected from one field of view, as illustrated as an example as 1102B.Part (d) of FIG. 11 illustrates a zoomed-in view of the image in Part(c) of FIG. 11 . Compared to the cultured cells, the mouse brain tissueshows significantly higher density of cells and mRNA spots. Furthermore,background signal inherent in the tissue makes imaging and spotdetection relatively more challenging compared to the cultured cells.Table 2 summarizes the results of Parts (a), (b), (c), and (d) of FIG.11 . The example illustrated in FIG. 11 shows that the particledetection method described in conjunction with FIGS. 8-9 is capable ofdetecting presence of mRNA segments in tissue samples or well plates.

TABLE 2 Comparison of spot counting results for two different types ofsingle molecule mRNA FISH samples U2OS cells on Mouse brain Sample typea 96 well plate tissue Gene target EGFR N/A Fluorescent dye Q670 Cy5 #of cells per FOV 146 275 # of spots per FOV 3,492 36,341 Average # ofspots per cell 24 132 Computation time 60 sec 198 sec

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs for amethod and an apparatus for synthetic aperture optics. Thus, whileparticular embodiments and applications of the present invention havebeen illustrated and described, it is to be understood that theinvention is not limited to the precise construction and componentsdisclosed herein and that various modifications, changes and variationswhich will be apparent to those skilled in the art may be made in thearrangement, operation and details of the method and apparatus of thepresent invention disclosed herein without departing from the spirit andscope of the invention as defined in the appended claims.

What is claimed is:
 1. A method for detecting particles on a target, themethod comprising the steps of: for a region of the target, illuminatingthe target with a plurality of structured illumination patterns togenerate a first series of raw intensity values for the region of thetarget, each structured illumination pattern characterized by a spatialfrequency and an illumination phase, wherein each raw intensity value inthe first series of raw intensity values is obtained by measuring acorresponding optical signal from the region of the target illuminatedwith a corresponding structured illumination pattern; generating a firstestimate that indicates whether a particle is present at the region ofthe target from the first series of raw intensity values; for anotherregion of the target, illuminating the target with the plurality ofstructured illumination patterns to generate a second series of rawintensity values for the another region of the target, wherein each rawintensity value in the second series of raw intensity values is obtainedby measuring a corresponding optical signal from the another region ofthe target illuminated with a corresponding structured illuminationpattern; and generating a second estimate that indicates whether theparticle or another particle is present at the another region of thetarget from the second series of raw intensity values.
 2. The method ofclaim 1, further comprising: after generating the first estimate, forthe region of the target, illuminating the target with the plurality ofstructured illumination patterns to generate a third series of rawintensity values for the region of the target, wherein each rawintensity value in the third series of raw intensity values is obtainedby measuring a corresponding optical signal from the region of thetarget illuminated with a corresponding structured illumination pattern;and generating a third estimate that indicates whether the particle ispresent at the region of the target from the third series of rawintensity values.
 3. The method of claim 1, wherein generating the firstestimate comprises: determining a first modulation score indicating adegree of variation in the first series of raw intensity values; andgenerating the first estimate for the region of the target by comparingthe first modulation score to a predetermined threshold.
 4. The methodof claim 3, wherein determining the first modulation score comprisescalculating a variance of two or more of the first series of rawintensity values, calculating a range of two or more of the first seriesof raw intensity values, calculating the variance of two or more of thefirst series of raw intensity values normalized by a mean of the two ormore of the first series of raw intensity values, or a goodness-of-fitmetric to a reference curve of two or more of the first series of rawintensity values.
 5. The method of claim 3, wherein determining thefirst modulation score comprises: determining a set of sub-modulationscores, each sub-modulation score determined by combining a respectivesubset of the first series of raw intensity values; and combining theset of sub-modulation scores to determine the first modulation score. 6.The method of claim 5, wherein the respective subset of the first seriesof raw intensity values are obtained by illuminating the region of thetarget with a corresponding subset of the plurality of structuredillumination patterns having a same spatial frequency.
 7. The method ofclaim 1, wherein the plurality of structured illumination patterns areselective excitation patterns.
 8. The method of claim 1, wherein theparticle is a biological molecule including at least one of DNAsegments, mRNA segments, or lncRNA.
 9. The method of claim 1, furthercomprising: generating a reconstructed image by processing at least thefirst series of raw intensity values, the reconstructed image includinga set of reconstructed intensity values obtained from processing the atleast the first series of raw intensity values; for the region of thetarget, determining a third estimate that indicates whether the particleis present at the region of the target based on one or morereconstructed intensity values corresponding to the region of thetarget; and generating a combined estimate for the region of the targetby comparing the first estimate to the third estimate.
 10. The method ofclaim 1, further comprising responsive to determining that the particleis present at the region of the target based on the first estimate,determining that the same particle is present at the another region ofthe target based on the second estimate.
 11. The method of claim 1,wherein the detection module is further configured to: responsive todetermining that the particle is present at the region of the targetbased on the first estimate, determine that the same particle is presentat the another region of the target based on the second estimate.
 12. Asystem for detecting particles on a target, the system comprising: aplurality of illumination modules configured to illuminate a region ofthe target with a plurality of structured illumination patterns that areeach characterized by a spatial frequency and an illumination phase, andat a later time, illuminate another region of the target with theplurality of structured illumination patterns, an optical imaging moduleconfigured to generate a first series of raw intensity values for theregion of the target, wherein each raw intensity value in the firstseries of raw intensity values is obtained by measuring a correspondingoptical signal from the region of the target illuminated with acorresponding structured illumination pattern, and generate a secondseries of raw intensity values for the another region of the target,wherein each raw intensity value in the second series of raw intensityvalues is obtained by measuring a corresponding optical signal from theanother region of the target illuminated with a corresponding structuredillumination pattern; and a detection module configured to generate, forthe region of the target, a first estimate that indicates whether aparticle is present at the region of the target from the first series ofraw intensity values, and generate, for the another region of thetarget, a second estimate that indicates whether the particle or anotherparticle is present at the another region of the target from the secondseries of raw intensity values.
 13. The system of claim 12, wherein theplurality of illumination modules is further configured to: aftergenerating the first estimate, for the region of the target, illuminatethe target with the plurality of structured illumination patterns togenerate a third series of raw intensity values for the region of thetarget, wherein each raw intensity value in the third series of rawintensity values is obtained by measuring a corresponding optical signalfrom the region of the target illuminated with a correspondingstructured illumination pattern, and wherein the detection module isfurther configured to: generate a third estimate that indicates whetherthe particle is present at the region of the target from the thirdseries of raw intensity values.
 14. The system of claim 12, wherein thedetection module is further configured to: determine a first modulationscore indicating a degree of variation in the first series of rawintensity values; and generate the first estimate for the region of thetarget by comparing the first modulation score to a predeterminedthreshold.
 15. The system of claim 14, wherein the first modulationscore is determined by calculating a variance of two or more of thefirst series of raw intensity values, calculating a range of two or moreof the first series of raw intensity values, calculating the variance oftwo or more of the first series of raw intensity values normalized by amean of the two or more of the first series of raw intensity values, ora goodness-of-fit metric to a reference curve of two or more of thefirst series of raw intensity values.
 16. The system of claim 14,wherein the detection module is further configured to: determine a setof sub-modulation scores, each sub-modulation score determined bycombining a respective subset of the first series of raw intensityvalues; and combine the set of sub-modulation scores to determine thefirst modulation score.
 17. The system of claim 16, wherein therespective subset of the first series of raw intensity values areobtained by illuminating the region of the target with a correspondingsubset of the plurality of structured illumination patterns having asame spatial frequency.
 18. The system of claim 12, wherein theplurality of structured illumination patterns are selective excitationpatterns.
 19. The system of claim 12, wherein the particle is abiological molecule including at least one of DNA segments, mRNAsegments, or lncRNA.
 20. The system of claim 12, further comprising: areconstruction module configured to generate a reconstructed image byprocessing at least the first series of raw intensity values, thereconstructed image including a set of reconstructed intensity valuesobtained from processing the at least the first series of raw intensityvalues, and wherein the detection module is further configured to: forthe region of the target, determine a third estimate that indicateswhether the particle is present at the region of the target based on oneor more reconstructed intensity values corresponding to the region ofthe target; and generate a combined estimate for the region of thetarget by comparing the first estimate to the third estimate.